Method and apparatus for displaying images in combination with taking images

ABSTRACT

An imaging apparatus. The apparatus includes a display means for displaying an image. The apparatus includes a camera means distributed throughout the display means to take images of objects in view of the display means. A screen. The screen includes a display for displaying an image. The screen includes a plurality of pinholes distributed in the display. The screen includes a plurality of sensors with at least one sensor of the plurality of sensors in alignment with and corresponding with one pinhole of the plurality of pinholes to receive light passing through the pinhole to image a ray of a specific size and a specific direction out from the display. A method for imaging.

FIELD OF THE INVENTION

[0001] The present invention is related to displays that have a camerapositioned to allow a viewer to look at the display and be seen aslooking directly at the camera. More specifically, the present inventionis related to a display having sensors distributed throughout thedisplay to form an image of a viewer looking at the display.

BACKGROUND OF THE INVENTION

[0002] A basic problem with video communication equipment is that thedisplay device and the imaging device (camera) are physically separated,eliminating the ability for direct, natural eye contact and transparentgaze awareness. Extensive work has been done researching this problemand proposing physical and/or compute based solutions (see referencesbelow). The need to both image and display is fundamental to manyactivities, including communicating, documenting, document scanning,security, etc.

[0003] This invention combines an electronic display with an imagingdevice in a way that permits the point of view of the imaging device tobe placed directly behind the display itself, enabling direct, naturaleye contact. In addition, the device has the ability to scan like aflatbed scanner, and functionally zoom, pan, tilt, and shift withoutmoving parts within the limits of its design, like a PTZ camera.

[0004] Prior art for the present invention includes displays withintegrated cameras, such as CRTs, notebook computers, PDAs, monitor topcameras, etc. Devices using partially silvered mirrors, projectors, CRTswith cameras integrated into the tube, and other approaches have beeninvestigated in the past, but did not have any integration of displayand imaging at the pixel level.

SUMMARY OF THE INVENTION

[0005] The present invention pertains to an imaging apparatus. Theapparatus comprises a display means for displaying an image. Theapparatus comprises a camera means distributed throughout the displaymeans to take images of objects in view of the display means.

[0006] The present invention pertains to a screen. The screen comprisesa display for displaying an image. The screen comprises a plurality ofpinholes distributed in the display. The screen comprises a plurality ofsensors with at least one sensor of the plurality of sensors inalignment with and corresponding with one pinhole of the plurality ofpinholes to receive light passing through the pinhole to image a ray ofa specific size and a specific direction out from the display.

[0007] The present invention pertains to a method for imaging. Themethod comprises the steps of applying a first image on a first portionof a screen. There is the step of receiving light from objects at asecond portion of the screen. The second portion is distributed in thefirst portion. There is the step of forming a second image from thelight from objects received at the second portion.

[0008] The present invention allows the creation of a combinationdisplay/imaging device which functions like a window does, allowing aremote “outside” person to look in, while allowing the person “inside”to see out. The combination of functions is natural and solves the wellknown eye gaze problem in visual communication. The design takesadvantage of electronic integration techniques to reduce part count,cost, and moving parts, and enables the development of display/imagedevices with very wide application areas.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the accompanying drawings, the preferred embodiment of theinvention and preferred methods of practicing the invention areillustrated in which:

[0010]FIG. 1 is a schematic representation of an apparatus of thepresent invention.

[0011]FIG. 2 is a schematic representation of a screen of the presentinvention.

[0012]FIG. 3 is an image taken by the apparatus.

DETAILED DESCRIPTION

[0013] Referring now to the drawings wherein like reference numeralsrefer to similar or identical parts throughout the several views, andmore specifically to FIG. 1 thereof, there is shown an imaging apparatus10. The apparatus 10 comprises a display means 12 for displaying animage. The apparatus 10 comprises a camera means 14 distributedthroughout the display means 12 to take images of objects in view of thedisplay means 12.

[0014] The present invention pertains to a screen 16, as shown in FIG.2. The screen 16 comprises a display 18 for displaying an image. Thescreen 16 comprises imaging means 11 for forming an images, such as aplurality of pinholes 20 or lenses distributed in the display 18. Thescreen 16 comprises a plurality of sensors 22 with at least one sensorof the plurality of sensors 22 in alignment with and corresponding withone pinhole of the plurality of pinholes 20 to receive light passingthrough the pinhole to image a ray of a specific size and a specificdirection out from the display 18.

[0015] Connected to each sensor is preferably image processing means 15having algorithms to enhance the image by addressing diffractionartifacts and light fall off. Preferably, the algorithms are embedded inthe screen and the data from the sensors are fed to them by connectionsbuilt into the screen, as is well known to one skilled in the art.

[0016] Preferably, the sensors 22 are single pixel sensors 22. Thesensors 22 are preferably arranged to converge to a point. Preferably,wherein the sensors 22 are arranged to define an effective point of viewbehind the display 18. There are preferably multiple sensors 22 inalignment with each pinhole to combine multiple rays of light behindeach panel. Preferably, each sensor is about 0.2 mm from the respectivepinhole.

[0017] The present invention pertains to a method for imaging. Themethod comprises the steps of applying a first image on a first portion24 of a screen 16. There is the step of receiving light from objects ata second portion 26 of the screen 16. The second portion 26 isdistributed in the first portion 24. There is the step of forming asecond image from the light from objects received at the second portion26.

[0018] Preferably, the forming step includes the step of converging raysof the light to a desired point. The receiving step preferably includesthe step of receiving the light at the second portion 26 having sensors22 distributed in the first portion 24. Preferably, the receiving stepincludes the step of receiving rays of light at the sensors 22 that passthrough pinholes 20 in the second portion 26.

[0019] In the operation of the invention, utilizing flat panel displaytechnology, such as organic polymer displays, integrate a large numberof single pixel (3 color) pinhole “cameras” across the display 18surface. By using IC fabrication techniques, fabricate and align thesingle pixel sensors 22 with accurately positioned and shaped pin holes,so that each pixel accurately images a ray of a specific size in aspecific direction out from the screen 16. By selecting and ordering aset of imaging pixels with rays that converge to a point, an image canbe generated. The effective point of view of this image is (typically)behind the panel, providing a more natural image without wide-angle viewdistortion, while maintaining a thin panel profile. Alternatively,lenses can be used instead of the pinholes. The lenses are formed in alike manner as the pinholes.

[0020] By having many such single pixel “cameras” and intelligentlydesigning the size, direction, and sensor distance of the pinholes 20,the angle of view (zoom), direction of view (pan and tilt), and positionof view (shift) can be controlled by selection and order of a subset ofpixels. Focus is fixed by the pinhole effect to be equal to theresolution of the selected set of pixels. By placing more than 1 pixel'ssensors 22 behind a pinhole, and varying the geometry of the sensors 22,multiple rays can be combined behind a single pinhole.

[0021] Use of the panel as a scanner is possible by using all pixels inphysical order due to the nearness of the imaged plane. The display 18light can be used to illuminate the paper or other object placed on thepanel. By scanning display 18 light across the panel, it is possible toincrease contrast and provide data appropriate for fingerprint/handrecognition. Some image processing may be necessary to deal with thesmall pixel size compared with the total screen 16 area.

[0022] By using the data from pixels of non-convergent rays, distanceand position information can be developed, providing a method for 3Dposition sensing.

[0023] Use of a light pen, where the pen device emits a known color ormodulated pattern, could be used to implement graphic tabletfunctionality. Changing the modulation of the pen could be used toindicate different color pens, varying pressure, the user's identity,etc.

[0024] Stereo imaging can be performed by appropriately designing andselecting the sensor pixel rays.

[0025] The convergence point for a set of rays can be behind or in frontof the panel, providing some control of effective point of view, andallowing a thin device close to the viewer to have a more “comfortable”point of view well behind the screen 16.

[0026] Many other effects can be generated by using the data from asuperset of the pixels required for an image. The actual design of theimaging pixel patterns would be somewhat application specific, withtradeoffs being made for cost, resolution, zoom/pan/tilt/shift range,and special functions such as stereo imaging and 3D position sensing.

[0027] The basic geometry of a single pixel pinhole camera, assuming a640×480 resolution, square pixels, and a 70° horizontal angle of view,requires 640/70°=0.109° per pixel. With the size of an RGB CCD elementset at about 0.01 mm (0.007 for each color in current CCDs), the pinholeneeds to be about 0.2 mm in front of the sensors 22, which is practicalto implement. The size of the pinhole would need to be determined forcorrect exposure and diffraction tradeoffs for a given application.There needs to be no reflection generated from the sides of the pathbetween the pinhole and the sensor. Refraction indices of the materialsinvolved would change these values slightly. The biggest issue is thediffraction affects from the small pinhole size. This must be dealt withor the geometry changed to achieve an optimal image.

[0028] The panel area consumed by the pinholes 20 is, ideally, theentire surface. This would eliminate the mismatch between the ray imagedby a single element and the equivalent ray from a traditional cameralocated at the convergence point. Note that a panel of nothing but holesis a window. The practical requirement of using a small portion of thisarea, (so that the display 18 retains sufficient image quality),introduces some tradeoffs. The rays that are imaged by a series ofpinholes 20 spaced around the panel is somewhat different than theoptimal view from the convergence point. If the distance to the imagedobject is known, this can be dealt with, otherwise there will be anun-imaged gridline region close to the screen 16 and overlapping pixelsfarther away. A compromise can be reached here based on the practicaluse distance of the screen 16, and note that the un-imaged gridlines aresimilar to the coverage of the sensors 22 in a traditional CCD arraywith a focused image. Pinholes 20 that are significantly smaller than adisplay 18 pixel (<0.01″) would not significantly affect the viewabilityof a 100 DPI display 18. Also of interest is the transparencycharacteristic of the newer organic display 18 technologies, resultingin placing an imaging panel behind the display 18 panel andsynchronizing image capture with display 18 pixel blanking.Micro-machine techniques to adjust the image characteristics can also beused.

[0029] Image quality of pinhole cameras is limited by the tradeoffbetween the diffraction of a small pinhole vs. the geometric resolutionimposed by the pinhole size. Unlike film, in the electronic domain,there is the ability to work with the data that comes from the sensors22 in real time, doing a convolution or other filtering to compensatefor the far-field (Fresnel) and near-field (Fraunhofer) diffractionaffects. If necessary, this can be done for multiple frequencies basedon the 3 color detectors, compensating or using the positional offset ofthe triplet sensors 22 to improve the image further.

[0030] Images were taken with a small CCD board camera with a ¼″ CCDarray was used. An image was taken with the supplied 3.8 mm lens, and aroughly corresponding image was taken using a good quality 100 micron(0.004″) pinhole. This pinhole diameter is optimal for a 2-3 mm focallength. The image was generated by handholding the pinhole above the CCDsensor, with minimal shielding of the sensor from stray light, whichreduced the contrast somewhat. FIG. 3 shows the resulting image taken bythe array.

[0031] Interconnection and interfacing to the imaging pixels is done bya scanning technique similar to existing CCDs. However, the potentialfor multiple megapixels will require extensions to enable real timevideo performance. This is accomplished by parallel scanning severalrows at a time, smart grouping of pixels into subsets that are not usedat the same time, or an addressing technique. Interface from the panelto the host should not be burdened with the extraneous pixel data. Useof an FPGA device utilizing Adaptive Computing techniques to select,order, and otherwise process the data coming from the sensors 22 allowsfull flexibility and performance to be maintained. An LVDS electricalinterface to the host is provided for interconnect, in parallel to thedisplay 18 drive interface.

[0032] The image processing addresses diffraction artifacts and lightfall off due to each lens or pinhole. The image processing is well knownto one skilled in the art for this purpose as is well known to oneskilled in the art. The imaging data from each sensor is enhanced bywell known image processing algorithms. The algorithms themselves arepreferably implanted in a part of the screen itself, which is reallyjust a very large chip.

[0033] Although the invention has been described in detail in theforegoing embodiments for the purpose of illustration, it is to beunderstood that such detail is solely for that purpose and thatvariations can be made therein by those skilled in the art withoutdeparting from the spirit and scope of the invention except as it may bedescribed by the following claims.

What is claimed is:
 1. An imaging apparatus comprising: a display meansfor displaying an image; and a camera means distributed throughout thedisplay means to take images of objects in view of the display means. 2.A screen comprising: a display for displaying an image; a plurality ofimaging means for forming an image distributed in the display; and aplurality of sensors with at least one sensor of the plurality ofsensors in alignment with and corresponding with one pinhole of theplurality of pinholes to receive light passing through the pinhole toimage a ray of a specific size and a specific direction out from thedisplay.
 3. A screen as described in claim 2 wherein each imaging meansis a pinhole.
 4. A screen as described in claim 3 wherein the sensorsare single pixel sensors.
 5. A screen as described in claim 4 whereinthe sensors are arranged to converge to a point.
 6. A screen asdescribed in claim 5 wherein the sensors are arranged to define aneffective point of view behind the display.
 7. A screen as described inclaim 6 wherein there are multiple sensors in alignment with eachpinhole to combine multiple rays of light behind each panel.
 8. A screenas described in claim 7 wherein each sensor is about 0.2 mm from therespective pinhole.
 9. A screen as described in claim 2 wherein eachimaging means is a lens.
 10. A screen as described in claim 4 includingimage processing means connected to each sensor for enhancing the image.11. A screen as described in claim 10 wherein the image processing meanscompensates for diffraction artifacts and light fall off.
 12. A screenas described in claim 11 wherein the image processing means is incontact with the display.
 13. A method for imaging comprising the stepsof: applying a first image on a first portion of a screen; receivinglight from objects at a second portion of the screen, the second portiondistributed in the first portion; and forming a second image from thelight from objects received at the second portion.
 14. A method asdescribed in claim 13 wherein the forming step includes the step ofconverging rays of the light to a desired point.
 15. A method asdescribed in claim 14 wherein the receiving step includes the step ofreceiving the light at the second portion having sensors distributed inthe first portion.
 16. A method as described in claim 15 wherein thereceiving step includes the step of receiving rays of light at thesensors that pass through pinholes in the second portion.